DAQ: support for FRD file header with file locking #30206

smorovic · 2020-06-12T08:49:57Z

PR description:

raw (FRD) file format for DAQ3 has recently been modified to include metadata in the header, so json files can be removed (due to 2 MB block constraint in huge-page ramdisk). Initially support was added with file broker mode. Now it is added to the file-locking mode which is used in testing and in Hilton.
Support for the file header is also added to the single-buffer mode.
Error handling in file locking mode is improved. It also allows unit test to catch problem in case JSON (raw) files can not be found. Since throwing exception is an issue in separate file discovery thread used by input source, error condition is propagated to the main thread instead of throwing directly.

PR includes also two bug fixes:

unique_ptr release() calls do not execute destructor of allocated object wrapped with unique_ptr. This created a (small) memory leak and also caused processed files to not be deleted in single-threaded input source mode (which is available, but not used by default in production setup).
in the check of non-opened fd value comparision was with 0, it should be compared with -1.

PR validation:

Verified to work correctly in DAQ test system in single-buffered/multi-buffered running modes, with file broker/locking, and with/without file header (and combinations).

calling release on unique_ptr instead of reset

…cking mode (Hilton and testing use case). includes bugfix: initial value of rawFd should be -1 when reading the file by the worker thread.

… source mode.

… now results in exception which allows unit test to fail in case of error

cmsbuild · 2020-06-12T08:50:24Z

The code-checks are being triggered in jenkins.

cmsbuild · 2020-06-12T08:55:35Z

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-30206/16057

This PR adds an extra 48KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File EventFilter/Utilities/interface/EvFDaqDirector.h modified in PR(s): [RFC] Test clang-tidy --checks performance-unnecessary-value-param #29933
- File EventFilter/Utilities/src/EvFDaqDirector.cc modified in PR(s): [RFC] Test clang-tidy --checks performance-unnecessary-value-param #29933, [RFC] Test clang-tidy --checks performance-faster-string-find #29926, [RFC] Test clang-tidy --checks modernize-loop-convert #29935, [RFC] Test clang-tidy --checks performance-for-range-copy #29927
- File EventFilter/Utilities/src/FedRawDataInputSource.cc modified in PR(s): [RFC] Test clang-tidy --checks modernize-make-unique #29934, [RFC] Test clang-tidy --checks modernize-use-auto #29936, [RFC] Test clang-tidy --checks performance-faster-string-find #29926, [RFC] Test clang-tidy --checks modernize-loop-convert #29935, [RFC] Test clang-tidy --checks modernize-use-emplace #29937

Code check has found code style and quality issues which could be resolved by applying following patch(s)

code-format:
https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-30206/16057/code-format.patch
e.g. curl https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-30206/16057/code-format.patch | patch -p1
You can also run scram build code-format to apply code format directly

cmsbuild · 2020-06-12T09:06:28Z

The code-checks are being triggered in jenkins.

cmsbuild · 2020-06-12T09:10:41Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-30206/16060

This PR adds an extra 48KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File EventFilter/Utilities/interface/EvFDaqDirector.h modified in PR(s): [RFC] Test clang-tidy --checks performance-unnecessary-value-param #29933
- File EventFilter/Utilities/src/EvFDaqDirector.cc modified in PR(s): [RFC] Test clang-tidy --checks performance-unnecessary-value-param #29933, [RFC] Test clang-tidy --checks performance-faster-string-find #29926, [RFC] Test clang-tidy --checks modernize-loop-convert #29935, [RFC] Test clang-tidy --checks performance-for-range-copy #29927
- File EventFilter/Utilities/src/FedRawDataInputSource.cc modified in PR(s): [RFC] Test clang-tidy --checks modernize-make-unique #29934, [RFC] Test clang-tidy --checks modernize-use-auto #29936, [RFC] Test clang-tidy --checks performance-faster-string-find #29926, [RFC] Test clang-tidy --checks modernize-loop-convert #29935, [RFC] Test clang-tidy --checks modernize-use-emplace #29937

cmsbuild · 2020-06-12T09:11:07Z

A new Pull Request was created by @smorovic (Srecko Morovic) for master.

It involves the following packages:

EventFilter/Utilities

@perrotta, @mommsen, @cmsbuild, @emeschi, @slava77 can you please review it and eventually sign? Thanks.
@Martin-Grunewald this is something you requested to watch as well.
@silviodonato, @dpiparo you are the release manager for this.

cms-bot commands are listed here

perrotta · 2020-06-12T11:14:21Z

please test

cmsbuild · 2020-06-12T11:14:46Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/6990/console Started: 2020/06/12 15:58

cmsbuild · 2020-06-12T15:12:12Z

+1
Tested at: 8c55286
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-92bfae/6990/summary.html
CMSSW: CMSSW_11_2_X_2020-06-12-1100
SCRAM_ARCH: slc7_amd64_gcc820

cmsbuild · 2020-06-12T15:12:14Z

Comparison job queued.

mommsen · 2020-06-12T15:32:49Z

+1

cmsbuild · 2020-06-12T17:16:31Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-92bfae/6990/summary.html

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 36
DQMHistoTests: Total histograms compared: 2778811
DQMHistoTests: Total failures: 1
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2778760
DQMHistoTests: Total skipped: 50
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 35 files compared)
Checked 152 log files, 16 edm output root files, 36 DQM output files

perrotta · 2020-06-15T15:56:40Z

+1

Updates relevant for DAQ
Jenkins tests pass

cmsbuild · 2020-06-15T15:57:05Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @silviodonato, @dpiparo (and backports should be raised in the release meeting by the corresponding L2)

silviodonato · 2020-06-16T07:55:50Z

+1

smorovic added 4 commits June 11, 2020 13:30

fix single-threaded file deletion and several small memory leaks when

47288cc

calling release on unique_ptr instead of reset

suppord RAW (FRD) file with header and no JSON file also with file lo…

3c73a3f

…cking mode (Hilton and testing use case). includes bugfix: initial value of rawFd should be -1 when reading the file by the worker thread.

Added raw FRD header support to single-buffered/single-threaded input…

ff3cd99

… source mode.

improve error handling for file-locking mode. Missing index(raw) file…

1a49eb5

… now results in exception which allows unit test to fail in case of error

cmsbuild added this to the CMSSW_11_2_X milestone Jun 12, 2020

cmsbuild added code-checks-pending comparison-pending daq-pending orp-pending pending-signatures reconstruction-pending tests-pending labels Jun 12, 2020

cmsbuild added code-checks-rejected and removed code-checks-pending labels Jun 12, 2020

code format

8c55286

cmsbuild added code-checks-pending and removed code-checks-rejected labels Jun 12, 2020

cmsbuild added code-checks-approved and removed code-checks-pending labels Jun 12, 2020

cmsbuild added tests-started and removed tests-pending labels Jun 12, 2020

cmsbuild added tests-approved and removed tests-started labels Jun 12, 2020

cmsbuild added daq-approved and removed daq-pending labels Jun 12, 2020

cmsbuild added comparison-available and removed comparison-pending labels Jun 12, 2020

cmsbuild added fully-signed reconstruction-approved and removed pending-signatures reconstruction-pending labels Jun 15, 2020

cmsbuild added orp-approved and removed orp-pending labels Jun 16, 2020

cmsbuild merged commit 733b9a6 into cms-sw:master Jun 16, 2020

smorovic mentioned this pull request Oct 1, 2020

DAQ: added 16bit event info (flags) to FRD format #31543

Merged

smorovic mentioned this pull request Oct 28, 2020

DAQ backport (11_1_X) #31964

Merged

smorovic deleted the 12X-unique_ptr_fix branch October 30, 2020 18:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DAQ: support for FRD file header with file locking #30206

DAQ: support for FRD file header with file locking #30206

smorovic commented Jun 12, 2020

cmsbuild commented Jun 12, 2020

cmsbuild commented Jun 12, 2020

cmsbuild commented Jun 12, 2020

cmsbuild commented Jun 12, 2020

cmsbuild commented Jun 12, 2020

perrotta commented Jun 12, 2020

cmsbuild commented Jun 12, 2020 •

edited

Loading

cmsbuild commented Jun 12, 2020

cmsbuild commented Jun 12, 2020

mommsen commented Jun 12, 2020

cmsbuild commented Jun 12, 2020

perrotta commented Jun 15, 2020

cmsbuild commented Jun 15, 2020

silviodonato commented Jun 16, 2020

DAQ: support for FRD file header with file locking #30206

DAQ: support for FRD file header with file locking #30206

Conversation

smorovic commented Jun 12, 2020

PR description:

PR validation:

cmsbuild commented Jun 12, 2020

cmsbuild commented Jun 12, 2020

cmsbuild commented Jun 12, 2020

cmsbuild commented Jun 12, 2020

cmsbuild commented Jun 12, 2020

perrotta commented Jun 12, 2020

cmsbuild commented Jun 12, 2020 • edited Loading

cmsbuild commented Jun 12, 2020

cmsbuild commented Jun 12, 2020

mommsen commented Jun 12, 2020

cmsbuild commented Jun 12, 2020

perrotta commented Jun 15, 2020

cmsbuild commented Jun 15, 2020

silviodonato commented Jun 16, 2020

cmsbuild commented Jun 12, 2020 •

edited

Loading